Anchor recognition in reality system

ABSTRACT

A control method, suitable for head-mounted devices located in a physical environment, includes following operations. Images of the physical environment are captured over time by the head-mounted devices. Candidate objects and object features of the candidate objects are extracted from the images. Local determinations are generated about whether each of the candidate objects is fixed or not. The object features and the local determinations are shared between the head-mounted devices. An updated determination is generated about whether each of the candidate objects is fixed or not according to the local determinations shared between the head-mounted devices.

RELATED APPLICATIONS

The present application is a Divisional Application of the U.S.application Ser. No. 15/863,994, filed Jan. 8, 2018, which is hereinincorporated by reference.

BACKGROUND Field of Invention

The disclosure relates to a reality system. More particularly, thedisclosure relates to how to recognize anchor points in a realitysystem.

Description of Related Art

Virtual Reality (VR), Augmented Reality (AR), Substitutional Reality(SR), and/or Mixed Reality (MR) devices are developed to provideimmersive experiences to users. When a user wearing a head-mounteddevice, the visions of the user will be covered by the immersive contentshown on the head-mounted device. The immersive content shows a scenarioof a specific space.

These reality systems usually equip a tracking component to locate userswearing the head-mounted device, so as to acknowledge a location or amovement of the user in the real world. The immersive content displayedon the head-mounted device will vary according to the location or themovement of the user, such that the user may have a better experience inthe VR, AR, SR or MR scenario.

SUMMARY

The disclosure provides a reality system including a first head-mounteddevice. The first head-mounted device is located in a physicalenvironment. The first head-mounted device includes a camera unit, acommunication unit and a processor. The camera unit is configured forcapturing images of the physical environment over time. Thecommunication unit is configured for communicating with secondhead-mounted devices. The processor is coupled to the camera unit andthe communication unit. The processor is configured to: extract firstcandidate objects and first object features of the first candidateobjects from the images captured by the camera unit; generate a firstlocal determination about whether each of the first candidate objects isfixed or not; transmit the first object features and the first localdetermination to the second head-mounted devices; receive second objectfeatures of second candidate objects and second local determinationsabout whether each of the second candidate objects is fixed or not fromthe second head-mounted devices; and generate an updated determinationabout whether each of the first candidate objects is fixed or notaccording to the first local determination and the second localdeterminations.

The disclosure also provides a reality system including head-mounteddevices and a server device. The head-mounted devices are located in aphysical environment. Each of the head-mounted devices is configured toextract candidate objects, extract object features of the candidateobjects and generate local determinations about whether each of thecandidate objects is fixed or not in view of each of the head-mounteddevices. The server device is communicated with the head-mounteddevices, wherein the server device is configured to: collect the objectfeatures of the candidate objects and the local determinations from thehead-mounted devices; generate an updated determination about whethereach of the candidate objects is fixed or not according to the localdeterminations collected from the head-mounted devices; and, transmitthe updated determination to each of the head-mounted devices.

The disclosure also provides a control method suitable for head-mounteddevices located in a physical environment. The control method includesfollowing operations. Images of the physical environment are capturedover time by the head-mounted devices. Candidate objects and objectfeatures of the candidate objects are extracted from the images. Localdeterminations are generated about whether each of the candidate objectsis fixed or not. The object features and the local determinations areshared between the head-mounted devices. An updated determination isgenerated about whether each of the candidate objects is fixed or notaccording to the local determinations shared between the head-mounteddevices.

It is to be understood that both the foregoing general description andthe following detailed description are by examples, and are intended toprovide further explanation of the invention as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure can be more fully understood by reading the followingdetailed description of the embodiment, with reference made to theaccompanying drawings as follows:

FIG. 1 is a schematic diagram illustrating a reality system according toan embodiment of this disclosure.

FIG. 2 is a functional block diagram illustrating the reality system inFIG. 1 according to an embodiment of the disclosure.

FIG. 3 is a flow diagram illustrating a control method suitable to beutilized on the head-mounted device in FIG. 1 and FIG. 2.

FIG. 4A to FIG. 4C are schematic diagrams illustrating views ofdifferent head-mounted devices in the physical environment.

FIG. 5 is a schematic diagram illustrating a head-mounted device newlyentered the physical environment in an embodiment of the disclosure.

FIG. 6 is a schematic diagram illustrating a reality system according toanother embodiment of the disclosure.

FIG. 7 is a functional block diagram illustrating the reality system inFIG.

FIG. 8 is a flow diagram illustrating a control method suitable for thereality system in FIG. 6.

DETAILED DESCRIPTION

Reference will now be made in detail to the present embodiments of thedisclosure, examples of which are illustrated in the accompanyingdrawings. Wherever possible, the same reference numbers are used in thedrawings and the description to refer to the same or like parts.

Reference is made to FIG. 1, which is a schematic diagram illustrating areality system 100 according to an embodiment of this disclosure. Thereality system 100 includes at least one head-mounted device located ina physical environment PE. In the embodiment shown in FIG. 1, thereality system 100 includes three head-mounted devices 110, 120 and 130,which are mounted on heads of different users in the physicalenvironment PE. However, the disclosure is not limited to threehead-mounted devices as shown in FIG. 1. The reality system 100 mayinclude two or more head-mounted devices. These users may locate atdifferent positions and face toward various directions in the physicalenvironment PE.

The reality system 100 is one of a Virtual Reality (VR), an AugmentedReality (AR), a Substitutional Reality (SR), or a Mixed Reality (MR)system. In order to provide an immersive experience to the users, thehead-mounted devices 110, 120 and 130 are configured to construct a mapof the physical environment PE and sense relative locations of thehead-mounted devices 110, 120 and 130 in the physical environment PE. Insome embodiments, Simultaneous Localization and Mapping (SLAM)technology is utilized by the reality system 100 to construct the map ofan unknown environment (e.g., the physical environment PE) whilesimultaneously tracking the head-mounted devices 110, 120 and 130 withinthe unknown environment.

As shown in FIG. 1, there are some candidate objects OBJ1-OBJ9 locatedat different positions in the physical environment PE. The map of thephysical environment PE can be constructed by observing the physicalenvironment PE with some sensors on the head-mounted devices 110, 120and 130. On the other hand, the relative locations of the head-mounteddevices 110, 120 and 130 in the physical environment PE can be sensed bycalculating distances between one head-mounted device and referencepoints in the physical environment PE. While locating the head-mounteddevices 110, 120 and 130 in the physical environment PE, finding thevalid reference points is important. In an ideal case, the referencepoints utilized to locate the head-mounted devices 110, 120 and 130 in alocating algorithm should be fixed at same positions over time. Sincethe reference points are fixed at the same positions in the physicalenvironment PE, the locations of the head-mounted devices 110, 120 and130 can be generated by measuring relative distances between thehead-mounted devices 110, 120 and 130 and the reference points. If thereference points are moved for some reasons, the locating resultsgenerated by the locating algorithm will be shifted and include errors.In this case, visions displayed to the users wearing the head-mounteddevices 110, 120 and 130 will change dramatically and inconsistently. Insome embodiments of this disclosure, the reality system 100 is able toextinguish fixed objects from the candidate objects OBJ1-OBJ9 in thephysical environment PE.

Reference is further made to FIG. 2, which is a functional block diagramillustrating the reality system 100 in FIG. 1 according to an embodimentof the disclosure. As shown in FIG. 2, the head-mounted device 110includes a camera unit 112, a processor 114 and a communication unit116. The camera unit 112 is configured for capturing images of thephysical environment PE over time. In some embodiments, the camera unit112 can include one camera component located at the front side of thehead-mounted device 110. In some other embodiments, the camera unit 112can include a dual camera module and/or a depth camera. Thecommunication unit 116 of the head-mounted device 110 is configured forcommunicating with other head-mounted devices 120 and 130. In anembodiment, the communication unit 116 can include a WiFi transceiver, aWiFi-Direct transceiver, a Bluetooth transceiver, a BLE transceiver, aZigbee transceiver and/or any equivalent wireless communicationtransceiver. In another embodiment, the communication unit 116 caninclude a wire connector (e.g., a USB connector, an Ethernet connector)for communication. The processor 114 is coupled to the camera unit 112and the communication unit 116. The processor 114 can be a centralprocessing unit, a graphic processing unit and/or a control integratedcircuit on the head-mounted device 110.

Similarly, the head-mounted device 120 includes a camera unit 122, aprocessor 124 and a communication unit 126, and the head-mounted device130 includes a camera unit 132, a processor 134 and a communication unit136. Each of the head-mounted devices 110, 120 and 130 includes similarcomponents and these components are configured to perform similarfunctions. For brevity reasons, the following embodiments willdemonstrate details of features on the head-mounted device 110. Thehead-mounted device 120 or 130 also includes corresponding details offeatures respectively.

Reference is further made to FIG. 3 and FIG. 4A. FIG. 3 is a flowdiagram illustrating a control method 300 suitable to be utilized on thehead-mounted device 110 in FIG. 1 and FIG. 2. FIG. 4A is a schematicdiagram illustrating the head-mounted device 110 in the physicalenvironment PE.

As shown in FIG. 2, FIG. 3 and FIG. 4A, operation S310 of the controlmethod 300 is performed to capture images IMG1 of the physicalenvironment PE over time by the camera unit 112. In some embodiments,the camera unit 112 is configured to capture images IMG1 of the physicalenvironment PE periodically (e.g., every 10 microseconds or every 100microseconds). The images IMG1 are sequential images to reflect apartial view of the physical environment PE in front of the head-mounteddevice 110. The partial view corresponds to a field of view (FOV) of thecamera unit 112. In some embodiments, the FOV of the camera unit 112 isunable to cover the whole scene of the physical environment PE. In someembodiments, images IMG1 captured by the camera unit 112 are utilized bythe processor 114 to construct a partial map of the physical environmentPE and locate the head-mounted device 110.

Operation S320 is performed by the processor 114 to extract candidateobjects from the images IMG1 captured by the camera unit 112 and alsoextract object features OF1 of the candidate objects from the imagesIMG1. As shown in FIG. 4A, the images IMG1 covers the candidate objectsOBJ1-OBJ5. Therefore, the processor is able to extract the candidateobjects OBJ1-OBJ5 and the object features OF1 of the candidate objectsOBJ1-OBJ5 from the images IMG1. In some embodiments, the object featuresOF1 include at least one of colors, sizes and shapes of the candidateobjects OBJ1-OBJ5. Operation S330 is performed by the processor 114 togenerate a local determination LD1 about whether each of the candidateobjects OBJ1-OBJ5 is fixed or not. An example of the object features OF1and the local determination LD1 generated by the processor 114 is listedin the following Table 1:

TABLE 1 OF1 LD1 color size shape fixed or not item1 red 40 cm * 50 cmrectangle Y (OBJ1) item2 brown 95 cm * 60 cm horizontal plane N (OBJ4)with supporters item3 black 30 cm * 30 cm square Y (OBJ2) item4 yellow50 cm * 35 cm rectangle Y (OBJ3) item5 blue  8 cm * 15 cm concavecylinder Y (OBJ5)

The local determination LD1 can be generated by the processor 114according to local information (e.g., the images IMG1 captured by thecamera unit 112 and/or the object features OF1 of the candidate objectsOBJ1-OBJ5). In an embodiment, the local determination LD1 is generatedby the processor 114 based on an object recognition algorithm. Forexample, the processor 114 recognizes the item1 in Table 1(corresponding to the candidate object OBJ1 in FIG. 4A) as a firehydrant based on the red color and the rectangle shape. The processor114 may recognize the item2 (corresponding to the candidate object OBJ4in FIG. 4A) as a table based on the brown color and the shape of ahorizontal plane with supporters. Similarly, the processor 114 mayfurther recognize the item3 (corresponding to the candidate object OBJ2)as a picture frame based on the size and the shape. The processor 114recognize the item4 (corresponding to the candidate object OBJ3 asanother picture frame. The processor 114 recognize the item5(corresponding to the candidate object OBJ5 as a vase based on the sizeand the shape. Based on aforesaid object recognition results, theprocessor 114 can determine whether the items in Table 1 are fixed ornot. In the embodiment, the item1, the item3 and the item4 (i.e.,candidate objects OBJ1, OBJ2 and OBJ3 shown in FIG.) are determined tobe fixed locally by the processor 114 on the head-mounted device 110. Insome cases, the fixed items can be regarded as anchor points in thephysical environment PE for locating the head-mounted device 110.However, the head-mounted device 110 is only capable to observe thepartial map of the physical environment PE from one vision point, suchthat it is hard to ensure the correctness of the local determinationLD1.

For example, the item5 corresponding to the candidate object OBJ5 is avase, which can be movable over time, such that the candidate objectOBJ5 is not suitable to be regarded as an anchor point for locatingfunction. In this case, the local information LD1 based on limitedinformation (only observing the candidate object OBJ5 for 3 seconds fromone vision point) may determine the candidate object OBJ5 to be fixed bymistake.

In addition, once the user move to some other positions or faces towarddifferent directions, the item 1, the item3 and the item4 (e.g., thecandidate objects OBJ1, OBJ2 and OBJ3) may not appear in the field ofview of the camera unit 112. The head-mounted device 110 will not ableto locate itself until new anchor points are found.

As shown in FIG. 2 and FIG. 3, operation S340 is performed by theprocessor 114 to transmit the object features OF1 and the localdetermination LD1 (i.e., as shown in Table 1) through the communicationunit 116 to other head-mounted devices 120 and 130. Operation S350 isperformed by the processor 114 to receive object features OF2 and localdetermination LD2 from the head-mounted devices 120 and receive objectfeatures OF3 and local determination LD3 from the head-mounted devices130.

Reference is further made to FIG. 4B, which is a schematic diagramillustrating the head-mounted device 120 in the physical environment PE.Similar to operations S310-S330 in aforesaid embodiments, thehead-mounted devices 120 is able to capture images IMG2 as shown in FIG.4B of the physical environment over time. The images IMG2 captured bythe camera unit 122 of the head-mounted device 120 may be different fromthe images IMG1 because the head-mounted device 120 is located atdifferent position from the head-mounted devices 110.

In this embodiment shown in FIG. 4B, the images IMG2 covers anotherpartial view of the physical environment PE. The image IMG2 coverscandidate objects OBJ1, OBJ5, OBJ6, OBJ7, OBJ8 and OBJ9. Similarly, theprocessor 124 will extract candidate objects from the images IMG2captured by the camera unit 122, extract object features OF2 of thecandidate objects OBJ1 and OBJ5-OBJ9 from the images IMG2, and generatea local determination LD2 about whether each of the candidate objectsOBJ1 and OBJ5-OBJ9 is fixed or not. An example of the object featuresOF2 and the local determination LD2 generated by the processor 124 islisted in the following Table 2:

TABLE 2 OF2 LD2 color size shape fixed or not item1 white 15 cm * 5 cm rectangle Y (OBJ9) item2 blue  8 cm * 15 cm concave N (OBJ5) cylinderitem3 green 40 cm * 50 cm rectangle Y (OBJ8) item4 white 30 cm * 30 cmcircle Y (OBJ6) item5 khaki 60 cm * 55 cm vertical and N (OBJ7)horizontal planes with supporters item6 red 40 cm * 50 cm rectangle Y(OBJ1)

Reference is further made to FIG. 4C, which is a schematic diagramillustrating the head-mounted device 130 in the physical environment PE.In this embodiment shown in FIG. 4C, the images IMG3 covers anotherpartial view of the physical environment PE. The image IMG3 coverscandidate objects OBJ1, OBJ2, OBJ3, OBJ5 and OBJ6. Similarly, theprocessor 134 will extract candidate objects from the images IMG3captured by the camera unit 132, extract object features OF3 of thecandidate objects OBJ1-OBJ3 and OBJ5-OBJ6 from the images IMG3, andgenerate a local determination LD3 about whether each of the candidateobjects OBJ1-OBJ3 and OBJ5-OBJ6 is fixed or not. An example of theobject features OF3 and the local determination LD3 generated by theprocessor 134 is listed in the following Table 3:

TABLE 3 OF3 LD3 color size shape fixed or not item1 red 40 cm * 50 cmrectangle N (OBJ1) item2 blue  8 cm * 15 cm concave N (OBJ5) cylinderitem3 green 40 cm * 50 cm rectangle Y (OBJ3) item4 white 30 cm * 30 cmcircle Y (OBJ6) item3 black 30 cm * 30 cm square Y (OBJ2)

After the head-mounted device 110 receives the object features OF2 andthe local determination LD2 from the head-mounted devices 120 and thereceive object features OF3 and the local determination LD3 from thehead-mounted devices 130, operation S360 is performed by the processor114 of the head-mounted device 110 to generate an updated determinationabout whether each of the candidate objects is fixed or not according tothe local determination LD1 and the local determinations LD2 and LD3received from other head-mounted devices 120 and 130.

Based on aforesaid Table 1, Table 2 and Table 3, the updateddetermination on the head-mounted devices 110 can be generated bycomparing the object features OF1 in Table 1 and the object features OF2and OF3 in Table 2 and Table 3, so as to map a correspondencerelationship between the items in different tables. In an embodiment,the head-mounted devices 110, 120, 130 generate Table 1, Table 2 andTable 3 individually, and the items in these tables are not necessaryranked in the same sequence. Therefore, the head-mounted device 110refers the object features OF2 and OF3 in Table 2 and Table 3 to alignthe items from Table 2 and Table 3 to suitable items in Table 1. Forexample, the item6 in Table 2 having a color of red, a size of 40 cm*50cm and a shape of rectangle matches to the item1 in Table 1 on thehead-mounted device 110. In this case, the head-mounted device 110 willmap the item6 in Table 2 to the item1 in Table 1, and combine the localdetermination LD2 about the item6 in Table 2 into a corresponding columnin the updated determination on the head-mounted devices 110. The item1in Table 3 also having a color of red, a size of 40 cm*50 cm and a shapeof rectangle matches to the item1 in Table 1 on the head-mounted device110. In this case, the head-mounted device 110 will map the item1 inTable 3 to the item1 in Table 1, and combine the local determination LD3about the item1 in Table 3 into a corresponding column in the updateddetermination on the head-mounted devices 110.

Similarly, each of the items in Table 2 and Table 3 can be mapped towarditems in Table 1 by processor 114. An example of the updateddetermination on the head-mounted devices 110 is listed in the followingTable 4:

TABLE 4 OF1 LD1 LD2 LD3 UD color size shape Fixed? Fixed? Fixed? Fixed?item1 red 40 cm * rectangle Y Y N Y (OBJ1) 50 cm item2 brown 95 cm *horizontal N — — N (OBJ4) 60 cm plane with supporters item3 black 30cm * square Y — Y Y (OBJ2) 30 cm item4 yel- 50 cm * rectangle Y — Y Y(OBJ3) low 35 cm item5 blue  8 cm * concave Y N N N (OBJ5) 15 cmcylinder item6 white 30 cm * circle — Y Y Y (OBJ6) 30 cm item7 khaki 60cm * . . . — N — N (OBJ7) 55 cm . . . . . . . . . . . . . . . . . . . .. . . .

In an embodiment, if some items in Table 2 or Table 3 fail to find amatch in existed items in Table 1, the head-mounted device 110 willmerge the unmatched items from Table 2 and Table 3 into a new datacolumn in Table 4. For example, the item6 and the item7 are not originalitems existed in Table 1.

As shown in Table 4, the item1 corresponding to the candidate objectOBJ1 is an overlapping object found in Table 1, Table 2 and Table 3,such that the updated determination UD relative to the candidate objectOBJ1 in Table 4 is decided by all of the local determinations LD1, LD2and LD3. The candidate object OBJ1 is determined to be fixed in two ofthe local determinations LD1 and LD2, and determined to be not fixed inone local determination LD3. In this case, the updated determination UDwill decide that the candidate object OBJ1 is fixed, because there aretwo head-mounted devices 110 and 120 of the opinion that the candidateobject OBJ1 is fixed against one head-mounted device 130.

As shown in Table 4, the item5 corresponding to the candidate objectOBJ5 is an overlapping object found in Table 1, Table 2 and Table 3,such that the updated determination UD relative to the candidate objectOBJ5 in Table 4 is decided by all of the local determinations LD1, LD2and LD3. The candidate object OBJ5 is determined to be not fixed in twoof the local determinations LD2 and LD3, and determined to be fixed inone local determination LD1. In this case, the updated determination UDwill decide that the candidate object OBJ1 is not fixed, because thereare two head-mounted devices 120 and 130 of the opinion that thecandidate object OBJ1 is not fixed against one head-mounted device 110.

As shown in Table 4, the item3 corresponding to the candidate objectOBJ2 is an overlapping object found in Table 1 and Table 3, such thatthe updated determination UD relative to the candidate object OBJ2 inTable 4 is decided by the local determinations LD1 and LD3. Thecandidate object OBJ2 is determined to be fixed in two of the localdeterminations LD1 and LD3. In this case, the updated determination UDwill decide that the candidate object OBJ2 is fixed.

As shown in Table 4, the item2 corresponding to the candidate objectOBJ4 is a non-overlapping object found only in Table 1, such that theupdated determination UD relative to the candidate object OBJ4 in Table4 is decided by the local determination LD1. As shown in Table 4, theitem7 corresponding to the candidate object OBJ7 is a non-overlappingobject found only in Table 2, such that the updated determination UDrelative to the candidate object OBJ7 in Table 4 is decided by the localdetermination LD2.

In aforesaid embodiment, the item6 and the item7 in Table 4corresponding to the candidate object OBJ6 and OBJ7 are not in the fieldof view of the camera unit 112 of the head-mounted device 110. Theseitem6 and the item7 corresponding to the candidate objects OBJ6 and OBJ7are established according to information received from otherhead-mounted devices. However, the camera unit 112 of the head-mounteddevice 110 currently has no visions to these objects OBJ6 and OBJ7. Asshown in FIG. 4A, the field of view of the camera unit 112 covers thecandidate objects OBJ1-OBJ5. Based on the updated determination UD, thecandidate objects OBJ1, OBJ2 and OBJ3 among the candidate objectsOBJ1-OBJ5 are fixed. The candidate objects OBJ1-OBJ3 determined to befixed in the updated determination UD are regarded as anchor points inthe physical environment PE. The processor 114 is further configured tocalculate relative distances between the head-mounted device 110 andeach of the anchor points (OBJ1-OBJ3). Operation S370 in FIG. 3 areperformed by the processor 114 to locate the head-mounted device 110within the physical environment PE according to the relative distancesbetween the head-mounted device 110 and the fixed objects (OBJ1-OBJ3).

Based on aforesaid embodiment, the object features OF1-OF3 observed bydifferent head-mounted devices 110, 120 and 130 and the localdeterminations made by different head-mounted devices can be sharedamong the head-mounted devices 110, 120 and 130 within the physicalenvironment PE. In this case, each of the head-mounted devices 110, 120and 130 can acknowledge the opinion (about whether one object is fixedor not) from other head-mount devices located at different positions.The updated determination can be generated in accordance with allopinions observed from different positions, such that the correctness ofthe updated determination UD shall be higher than individual one of thelocal determination LD1, LD2 or LD3.

The head-mounted devices 120 and 130 can also perform correspondingoperations similar to the embodiment shown in FIG. 3 to benefit from theupdated determination UD and locate the head-mounted devices 120 and 130within the physical environment PE.

Furthermore, the user wearing the head-mounted device 110 may rotatinghis/her head to different directions or move to different positions,such that the field of view of the head-mounted device 110 may changeover time. In this case, if the user wearing the head-mounted device 110rotating his head to the right side, the head-mounted device 110 willcapture the image covering the candidate objects OBJ6 and OBJ7. Based onthe updated determination UD shown in Table 4, the head-mounted device110 may acknowledge that the candidate objects OBJ6 can be utilized asan anchor point and the candidate object OBJ7 is not suitable to be ananchor point. The head-mounted device 110 can compare object features ofnewly captured images with Table 4. Once the object features of newlycaptured images found the match items (corresponding to the candidateobjects OBJ6 and OBJ7) in Table 4, the head-mounted device 110 is abledecide whether the candidate objects OBJ6 and OBJ7 are fixed or notimmediately. Therefore, the head-mounted device 110 is able to find newanchor point faster after the user rotating his/her head.

On the other hand, the object features OF1 of the candidate objectsregarded as the anchor points in view of the camera unit 112 of thehead-mounted device 110 can be transmitted from the head-mounted device110 to another head-mounted device 120 or 130. The head-mounted device120 or 130 is configured to recognize the anchor points in the physicalenvironment PE once the anchor points appeared in view of the cameraunit 122 or 132 of the head-mounted device 120 or 130.

In an embodiment as shown in FIG. 4A, the processor 114 of thehead-mounted device 110 is configured to measure a relative distance D1between a pair of the candidate objects OBJ1 and OBJ2. It is noticedthat the relative distance D1 measured by the head-mounted device 110may include a measurement error. For example, if the objects OBJ1 andOBJ2 gapped with a real distance of 50 cm in the physical environmentPE, the relative distance D1 measured on the head-mounted device 110 maybe 48 cm due to the measurement error. As shown in FIG. 4C, the pair ofcandidate objects OBJ1 and OBJ2 also covered by the images IMG3 capturedby the head-mounted device 130. A relative distance D2 between the pairof the candidate objects OBJ1 and OBJ2 is measured by the head-mounteddevice 130 in view of the camera unit 132. The relative distance D2measured by the head-mounted device 130 may also include anothermeasurement error.

In this case, the head-mounted device 110 is able to receive therelative distance D2 measured in view of the head-mounted device 130between the pair of the candidate objects OBJ1 and OBJ2. Thehead-mounted device 110 calculate an average relative distance betweenthe pair of the candidate objects OBJ1 and OBJ2 according to therelative distance D1 and the relative distance D2 measured by twohead-mounted device 130. For example, if the relative distance D2measured on the head-mounted device 130 may be 49 cm due to themeasurement error, the average relative distance will be 48.5 cm. Forexample, if the relative distance D2 measured on the head-mounted device130 may be 52 cm due to the measurement error, the average relativedistance will be 50 cm. By calculating an average of the relativedistances measured among different head-mounted devices, the measurementerror may be reduced.

Reference is further made to FIG. 5, which is a schematic diagramillustrating a head-mounted device 140 newly entered the physicalenvironment PE in an embodiment of the disclosure. The object featuresOF1 of the candidate objects OBJ1-OBJ3 regarded as the anchor points aretransmitted from the head-mounted device 110 to the head-mounted device140 newly entered the physical environment PE. In this case, thehead-mounted device 140 is configured to recognize the anchor points inthe physical environment PE according to the object features OF1 of thecandidate objects OBJ1-OBJ3 regarded as the anchor points. Similarly,the head-mounted device 140 may also receive the object features OF2/OF3of the candidate objects regarded as the anchor points from thehead-mounted devices 120 and 130. In this case, the head-mounted device140 newly entered the physical environment PE will recognize validanchor points easier and faster than observing the physical environmentPE by itself.

In aforesaid embodiments of FIG. 1 to FIG. 4, the object featuresOF1-OF3 and the local determinations LD1-LD3 are shared among thehead-mounted devices 110-130, each of the head-mounted devices isconfigured to generate the updated determination UD. However, thedisclosure is not limited thereto.

Reference is further made to FIG. 6 and FIG. 7. FIG. 6 is a schematicdiagram illustrating a reality system 200 according to anotherembodiment of the disclosure. FIG. 7 is a functional block diagramillustrating the reality system 200 in FIG. 6. The reality system 200 inFIG. 6 and FIG. 7 include head-mounted devices 210, 220 and 230 and aserver device 250. The head-mounted devices 210, 220 and 230 are locatedin a physical environment PE. As shown in FIG. 7, the head-mounteddevice 210 includes a camera unit 212, a processor 214 and acommunication unit 216. The head-mounted device 220 includes a cameraunit 222, a processor 224 and a communication unit 226. The head-mounteddevice 230 includes a camera unit 232, a processor 234 and acommunication unit 236. Components of the head-mounted devices 210-230are similar to the head-mounted device 110-130 in aforesaid embodimentshown in FIG. 2. The main difference between the reality system 200 inFIG. 6 and FIG. 7 and the reality system 100 in FIG. 1 and FIG. 2 isthat, the reality system 200 further includes the server device 250. Theserver device 250 is communicated with the head-mounted devices 210, 220and 230. The server device 250 includes a processor 254 and acommunication unit 256. The communication unit 256 can include a WiFitransceiver, a WiFi-Direct transceiver, a Bluetooth transceiver, a BLEtransceiver, a Zigbee transceiver and/or any equivalent wirelesscommunication transceiver. The processor 254 is coupled to thecommunication unit 256. The processor 254 can be a central processingunit, a graphic processing unit and/or a control integrated circuit onthe server device 250.

Reference is also made to FIG. 8, which is a flow diagram illustrating acontrol method 800, which is suitable to be utilized on the realitysystem 200 in FIG. 6 and FIG. 7.

Each of the head-mounted devices 210, 220 or 230 is configured tocapture images of the physical environment PE by the camera unit 212,222 or 232 respectively in operation S810 shown in FIG. 8. Each of thehead-mounted devices 210, 220 or 230 is configured to extract candidateobjects, extract object features OF1, OF2 or OF3 of the candidateobjects in operation S820 shown in FIG. 8. Each of the head-mounteddevices 210, 220 or 230 is configured to generate local determinationLD1, LD2 or LD3 about whether each of the candidate objects is fixed ornot in view of each of the head-mounted devices 210, 220 or 230 asoperation S830 shown in FIG. 8. Features about capture images of thephysical environment PE (S810) and how to extract object features OF1,OF2 or OF3 (S820) and generate local determinations LD1, LD2 or LD3(S830) on the head-mounted devices 210, 220 or 230 are similar tooperations S310, S320 and S330 disclosed in FIG. 3 and the head-mounteddevices 110, 120 or 130 in previous embodiments shown in FIG. 1, FIG. 2and FIG. 4A to FIG. 4C, and not repeated here.

In operation S840, each of the head-mounted devices 210, 220 or 230 isconfigured to transmit the object features OF1, OF2 or OF3 and the localdetermination LD1, LD2 or LD3 (referring to Table 1, Table 2 and Table 3in aforesaid embodiments) to the server device 250.

The server device 250 is communicated with the head-mounted devices210-230. The server device 250 is configured to collect the objectfeatures OF1-OF3 of the candidate objects and the local determinationsLD1-LD3 from the head-mounted devices 210-230 in operation S850. Theserver device 250 is configured to generate an updated determination UD(referring to Table 4 in aforesaid embodiments) about whether each ofthe candidate objects is fixed or not according to the localdeterminations LD1-LD3 collected from the head-mounted devices 210-230.The server device 250 is configured to transmit the updateddetermination UD (referring to Table 4 in aforesaid embodiments) to eachof the head-mounted devices 210-230, such that the head-mounted devices210-230 can locate positions of the head-mounted devices 210-230 withinthe physical environment PE relative to the fixed objects (e.g. anchorpoints) in the updated determination UD.

In an embodiment, the server device 250 is configured to generate theupdated determination by comparing the object features of the candidateobjects in views of the head-mounted devices 210-230 to map acorrespondence relationship between the candidate objects in views ofthe head-mounted devices 210-230. The server device 250 is configured togenerate the updated determination UD corresponding to an overlappedobject in view of the head-mounted devices according to the localdeterminations collected from the head-mounted devices. The serverdevice 250 is configured to generate the updated determination UDcorresponding to a non-overlapped object according to one of the localdeterminations LD1-LD3.

The object features OF1-OF3 includes at least one of colors, sizes andshapes of the candidate objects.

In an embodiment, the candidate objects determined to be fixed in theupdated determination UD are regarded by the server device as anchorpoints in the physical environment PE. Each of the head-mounted devices210-230 is further configured to calculate relative distances betweenthe head-mounted devices 210-230 and each of the anchor points. Based onthe relative distances, the processor 214, 224 or 234 is able to locatethe head-mounted device 210, 220 or 230 within the physical environmentPE.

In an embodiment, the object features of the candidate objects regardedas the anchor points are transmitted from the server device 250 to a newhead-mounted device (not shown in FIG. 6 and FIG. 7) newly entered thephysical environment PE. The new head-mounted device is configured torecognize the anchor points in the physical environment PE according tothe object features of the candidate objects regarded as the anchorpoints.

In an embodiment, the object features of the candidate objects regardedas the anchor points are transmitted from the server device 250 to allof the head-mounted devices 210, 220 and 230. The head-mounted devices210, 220 and 230 are further configured to measure local relativedistances between a pair of the candidate objects. The server device 250is configured to collect all of the local relative distances measured inview of the head-mounted devices 210-230 between the pair of thecandidate objects. The server device 250 is configured to calculate anaverage relative distance of all of the local relative distances. Eachof the head-mounted devices 210, 220 and 230 are further configured toobtain local measurement errors by comparing the local relativedistances and the average relative distance.

Based on aforesaid embodiment, the object features OF1-OF3 observed bydifferent head-mounted devices 210, 220 and 230 and the localdeterminations LD1-LD3 made by different head-mounted devices can becollected by the server device 250. The server device 250 can generatethe updated determination UD in accordance with aforesaid informationcollected from different head-mounted devices 210, 220 and 230. Theupdated determination UD can be generated in accordance with allopinions observed from different positions, such that the correctness ofthe updated determination UD shall be higher than individual one of thelocal determination LD1, LD2 or LD3.

Although the present invention has been described in considerable detailwith reference to certain embodiments thereof, other embodiments arepossible. Therefore, the spirit and scope of the appended claims shouldnot be limited to the description of the embodiments contained herein.

It will be apparent to those skilled in the art that variousmodifications and variations can be made to the structure of the presentinvention without departing from the scope or spirit of the invention.In view of the foregoing, it is intended that the present inventioncover modifications and variations of this invention provided they fallwithin the scope of the following claims.

What is claimed is:
 1. A reality system, comprising: a plurality ofhead-mounted devices located in a physical environment, wherein each ofthe head-mounted devices is configured to extract a plurality ofcandidate objects, extract a plurality of object features of thecandidate objects and generate a plurality of local determinations aboutwhether each of the candidate objects is fixed or not in view of each ofthe head-mounted devices; and a server device communicated with thehead-mounted devices, wherein the server device is configured to:collect the object features of the candidate objects and the localdeterminations from the head-mounted devices; comparing the objectfeatures to map a correspondence relationship between the candidateobjects collected from one of the head-mounted devices and the candidateobjects collected from others of the head-mounted devices to determinean overlapped object found in the candidate objects; generate an updateddetermination about whether each of the candidate objects is fixed ornot according to the local determinations collected from thehead-mounted devices, wherein the overlapped object is determined to befixed if a number of the local determinations corresponding to theoverlapped object determined to be fixed is greater than a number of thelocal determinations corresponding to the overlapped object determinedto be not fixed, wherein the overlapped object is determined to be notfixed if the number of the local determinations corresponding to theoverlapped object determined to be fixed is smaller than the number ofthe local determinations corresponding to the overlapped objectdetermined to be not fixed; and transmit the updated determination toeach of the head-mounted devices.
 2. The reality system of claim 1,wherein the server device is further configured to generate the updateddetermination by: generating the updated determination corresponding toa non-overlapped object according to one of the local determinations. 3.The reality system of claim 1, wherein the object features comprise atleast one of colors, sizes and shapes of the candidate objects.
 4. Thereality system of claim 1, wherein the candidate objects determined tobe fixed in the updated determination are regarded by the server deviceas anchor points in the physical environment, each of the head-mounteddevices is further configured to calculate a plurality of relativedistances between the head-mounted devices and each of the anchorpoints, and locate the head-mounted devices within the physicalenvironment according to the relative distances.
 5. The reality systemof claim 4, wherein the object features of the candidate objectsregarded as the anchor points are transmitted from the server device toa new head-mounted device newly entered the physical environment, thenew head-mounted device is configured to recognize the anchor points inthe physical environment according to the object features of thecandidate objects regarded as the anchor points.
 6. The reality systemof claim 4, wherein the object features of the candidate objectsregarded as the anchor points are transmitted from the server device toall of the head-mounted devices.
 7. The reality system of claim 1,wherein the head-mounted devices are further configured to measure localrelative distances between a pair of the candidate objects, and theserver device is configured to: collect all of the local relativedistances measured in view of the head-mounted devices between the pairof the candidate objects; and calculate an average relative distance ofall of the local relative distances.
 8. The reality system of claim 7,wherein each of the head-mounted devices are further configured toobtain local measurement errors by comparing the local relativedistances and the average relative distance.
 9. A control method for aserver device and a plurality of head-mounted devices located in aphysical environment, the control method comprising: extracting, by eachof the head-mounted devices, a plurality of candidate objects,extracting a plurality of object features of the candidate objects andgenerating a plurality of local determinations about whether each of thecandidate objects is fixed or not in view of each of the head-mounteddevices; collecting, by the server device, the object features of thecandidate objects and the local determinations from the head-mounteddevices; comparing the object features to map a correspondencerelationship between the candidate objects collected from one of thehead-mounted devices and the candidate objects collected from others ofthe head-mounted devices to determine an overlapped object found in thecandidate objects; generating, by the server device, an updateddetermination about whether each of the candidate objects is fixed ornot according to the local determinations collected from thehead-mounted devices, wherein the overlapped object is determined to befixed if a number of the local determinations corresponding to theoverlapped object determined to be fixed is greater than a number of thelocal determinations corresponding to the overlapped object determinedto be not fixed, wherein the overlapped object is determined to be notfixed if the number of the local determinations corresponding to theoverlapped object determined to be fixed is smaller than the number ofthe local determinations corresponding to the overlapped objectdetermined to be not fixed; and transmitting, by the server device, theupdated determination to each of the head-mounted devices.
 10. Thecontrol method of claim 9, further comprising: generating the updateddetermination corresponding to a non-overlapped object according to oneof the local determinations.
 11. The control method of claim 9, whereinthe object features comprise at least one of colors, sizes and shapes ofthe candidate objects.
 12. The control method of claim 9, furthercomprising: regarding the candidate objects determined to be fixed inthe updated determination as anchor points in the physical environment;and calculating a plurality of relative distances between thehead-mounted devices and each of the anchor points, and locating thehead-mounted devices within the physical environment according to therelative distances.
 13. The control method of claim 12, furthercomprising: transmitting, by the server device, the object features ofthe candidate objects regarded as the anchor points to a newhead-mounted device newly entered the physical environment; andrecognizing, by the new head-mounted device, the anchor points in thephysical environment according to the object features of the candidateobjects regarded as the anchor points.
 14. The control method of claim12, further comprising: transmitting, by the server device, the objectfeatures of the candidate objects regarded as the anchor points to allof the head-mounted devices.
 15. The control method of claim 9, furthercomprising: measuring local relative distances between a pair of thecandidate objects; collecting all of the local relative distancesmeasured in view of the head-mounted devices between the pair of thecandidate objects; and calculating an average relative distance of allof the local relative distances.
 16. The control method of claim 15,further comprising: obtaining local measurement errors by comparing thelocal relative distances and the average relative distance.